At Qure, we're building deep learning systems which help diagnose abnormalities from medical images. Most of the deep learning models are classification models which predict a probability of abnormality from a scan. However, just the probability score of the abnormality doesn't amount much to a radiologist if it's not accompanied by a visual interpretation of the model's decision. Interpretability of deep learning models is very much an active area of research and it becomes an even more crucial part of solutions in medical imaging. In this post, I'll be giving a brief overview of the different perturbation based techniques for deep learning based classification models and their drawbacks.